Object Segmentation and Tracking Using Video Locales

The ability to automatically locate and track objects from videos has always been very important in traditional applications such as surveillance, robotics, and object recognition. With the proliferation of digital videos and online multimedia data and the need of content-based multimedia encoding and retrieval, locating and tracking objects in digital videos become ever more important. This thesis presents a new technique based on feature localization for segmenting and tracking objects in videos. A video locale is a sequence of image feature locales that share similar features (color, texture, shape, and motion) in the spatio-temporal domain of videos. Image feature locales are grown from tiles (blocks of pixels) and can be non-disjoint and non-connected. Instead of using regions, the set of tiles belonging to the feature locale (called the envelope) is used to represent the locality of the feature. Intuitively, feature locales are significant feature blobs. To exploit the temporal redundancy in digital videos, two algorithms (intra-frame and inter-frame) are used to grow locales efficiently. Multiple motion tracking is achieved by tracking and performing tile-based dominant motion estimation for each locale separately (i.e., only member tiles are used); hence, the difficulty of multiple non-dominating motions is avoided and using tiles as the base unit makes the method more robust to pixel-level noise. Furthermore, video locales that move together through time may be grouped together to approximate multi-feature video objects. Being at a higher feature level than pixels and more robust than regions, video locales are more suited for content-based video processing. How video locales may be used in content-based multimedia encoding and retrieval such as outlined in MPEG-4 and MPEG-7 is discussed. Tests on natural videos have shown very good results.